Tracking arm movements to generate inputs for computer systems

ABSTRACT

A system including: a first sensor module having an inertial measurement unit and attached to an upper arm of a user, the first sensor module generating first motion data identifying an orientation of the upper arm; a second sensor module having an inertial measurement unit and attached to a hand of the user, the second sensor module generating second motion data identifying an orientation of the hand; and a computing device coupled to the first sensor module and the second sensor module through communication links, the computing device calculating, based on the orientation of the upper arm and the orientation of the hand, an orientation of a forearm connected to the hand by a wrist of the user and connected to the upper arm by an elbow joint of the user.

RELATED APPLICATIONS

The present application claims the benefit of the filing date of Prov.U.S. Pat. App. Ser. No. 62/507,085, filed May 16, 2017 and entitled“Methods, Systems, and Apparatuses for Calculating the Position of aSkeletal 3-D Model in Space using Predictive Interpolation Algorithms”,the entire disclosure of which is hereby incorporated herein byreference.

The present application relates to U.S. patent application Ser. No.15/492,915, filed Apr. 20, 2017 and entitled “Devices for ControllingComputers based on Motions and Positions of Hands”, which claims thebenefit of the filing dates of 62/325,925, filed Apr. 21, 2016 andentitled “Hand-Worn Devices for Controlling Computers based on Motionsand Positions of Hands and Fingers”, Prov. U.S. Pat. App. Ser. No.62/463,183, filed Feb. 24, 2017 and entitled “Devices for ControllingComputers based on Motions and Positions of Hands and Arms,” Prov. U.S.Pat. App. Ser. No. 62/463,209, filed Feb. 24, 2017 and entitled “Devicesfor Controlling Computers based on Motions and Positions of Hands,”Prov. U.S. Pat. App. Ser. No. 62/463,252, filed Feb. 24, 2017 andentitled “Devices for Controlling Computers based on Motions andPositions of Hands and Arms,” the entire disclosures of whichapplications are hereby incorporated herein by reference.

FIELD OF THE TECHNOLOGY

The embodiments disclosed herein relate to computer input devices ingeneral and more particularly but not limited to input devices forvirtual reality and/or augmented/mixed reality applications implementedusing computing devices, such as mobile phones, smart watches, similarmobile devices, and/or other devices.

BACKGROUND

U.S. Pat. App. Pub. No. 2014/0028547 discloses a user control devicehaving a combined inertial sensor to detect the movements of the devicefor pointing and selecting within a real or virtual three-dimensionalspace.

U.S. Pat. App. Pub. No. 2015/0277559 discloses a finger-ring-mountedtouchscreen having a wireless transceiver that wirelessly transmitscommands generated from events on the touchscreen.

U.S. Pat. App. Pub. No. 2015/0358543 discloses a motion capture devicethat has a plurality of inertial measurement units to measure the motionparameters of fingers and a palm of a user.

U.S. Pat. App. Pub. No. 2007/0050597 discloses a game controller havingan acceleration sensor and a gyro sensor. U.S. Pat. No. D772,986discloses the ornamental design for a wireless game controller.

Chinese Pat. App. Pub. No. 103226398 discloses data gloves that usemicro-inertial sensor network technologies, where each micro-inertialsensor is an attitude and heading reference system, having a tri-axialmicro-electromechanical system (MEMS) micro-gyroscope, a tri-axialmicro-acceleration sensor and a tri-axial geomagnetic sensor which arepackaged in a circuit board. U.S. Pat. App. Pub. No. 2014/0313022 andU.S. Pat. App. Pub. No. 2012/0025945 disclose other data gloves.

The disclosures of the above discussed patent documents are herebyincorporated herein by reference.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments are illustrated by way of example and not limitation inthe figures of the accompanying drawings in which like referencesindicate similar elements.

FIG. 1 illustrates a system to track arm movements according to oneembodiment.

FIG. 2 illustrates a system to control computer operations according toone embodiment.

FIG. 3 illustrates a skeleton model of an arm.

FIG. 4 illustrates the determination of the orientation of a forearmaccording to one embodiment.

FIG. 5 shows a method to compute the orientation of a forearm accordingto one embodiment.

FIG. 6 shows a detailed method to compute the orientation of a forearmaccording to one embodiment.

DETAILED DESCRIPTION

The following description and drawings are illustrative and are not tobe construed as limiting. Numerous specific details are described toprovide a thorough understanding. However, in certain instances, wellknown or conventional details are not described in order to avoidobscuring the description. References to one or an embodiment in thepresent disclosure are not necessarily references to the sameembodiment; and, such references mean at least one.

At least some embodiments disclosed herein allow arm movement trackingwithout a sensor device attached to the forearm. The forearm orientationis estimated, predicted, or computed from the orientation of the upperarm connected to the forearm and the orientation of the hand connectedto the forearm.

FIG. 1 illustrates a system to track arm movements according to oneembodiment.

In FIG. 1, an elbow joint (103) of a user connects an upper arm (101)and a forearm (109) of the user; and a wrist (107) connects the forearm(109) to a hand (105) of the user.

The orientation of the upper arm (101) is tracked/determined using anarm module (113) that is attached to the upper arm (101) via an armband(111). The orientation of the upper arm (101) is represented by a localcoordinate system X₁Y₁Z₁, where the lengthwise direction Y₁ is inparallel with the direction from the shoulder to the elbow joint (103),the direction X₁ is in parallel with the direction from the inner sideof the upper arm (101) to the outer side of the upper arm (101), and thedirection Z₁ is in parallel with the direction from the back side of theupper arm (101) to the front side of the upper arm (101).

The orientation of the hand (105) is tracked/determined using a handheldmodule (115). The orientation of the hand (105) is represented by alocal coordinate system X₃Y₃Z₃, where the lengthwise direction Y₃ is inparallel with the direction from the wrist (105) to the fingers, thedirection X₃ is in parallel with the direction from the back of the hand(105) to the palm of the hand (105), and the direction Z₃ is in parallelwith the direction from the edge of the palm to the thumb on the hand(105).

Preferably, the arm module (113) and the handheld module (115)separately report their motion/orientation parameters to a computingdevice (141) using wireless connections (117 and 119), such as apersonal area wireless network connection (e.g., Bluetooth connections),or a local area wireless network connection (e.g., Wi-Fi connections).

Alternatively, the arm module (113) may report its measurements to thehandheld module (115) (via a wired or wireless connection); and thehandheld module (115) communicates the motion/orientation measurementsto the computing device (141) (e.g., via a wired or wirelessconnection).

For example, the handheld module (115) and the arm module (113) can berespectively a base unit (or a game controller) and an arm/shouldermodule discussed in U.S. patent application Ser. No. 15/492,915, filedApr. 20, 2017 and entitled “Devices for Controlling Computers based onMotions and Positions of Hands”, the entire disclosure of whichapplication is hereby incorporated herein by reference.

At least some embodiments disclosed herein allow the orientation of theforearm (109) to be estimated, predicted, or calculated from theorientation of the hand (105) and the orientation of the upper arm (101)without the need for an additional sensor module to track theorientation of the forearm (109), as further discussed below.

FIG. 2 illustrates a system to control computer operations according toone embodiment. For example, the system of FIG. 2 can be implemented viaattaching the handheld module (115) and the arm module (113) to the hand(105) and the upper arm (101) respectively in a way illustrated in FIG.1.

In FIG. 2, the handheld module (115) and the arm module (113) havemicro-electromechanical system (MEMS) inertial measurement units (IMUs)(121 and 131) that measure motion parameters and determine orientationsof the hand (105) and the upper arm (101).

Each of the IMUs (131, 121) has a collection of sensor components thatenable the determination of the movement, position and/or orientation ofthe respective IMU along a number of axes. Examples of the componentsare: a MEMS accelerometer that measures the projection of acceleration(the difference between the true acceleration of an object and thegravitational acceleration); a MEMS gyroscope that measures angularvelocities; and a magnetometer that measures the magnitude and directionof a magnetic field at a certain point in space. In some embodiments,the IMUs use a combination of sensors in three and two axes (e.g.,without a magnetometer).

The computing device (141) has a motion processor (145), which includesa skeleton model (143) of the upper arm (101), the forearm (109), andthe hand (105) connected via the elbow joint (103) and the wrist (107)(e.g., illustrated FIG. 3). The motion processor (145) controls themovements of the corresponding parts of the skeleton model (143)according to the movements/orientations of the upper arm (101) and thehand (105) measured by the arm module (113) and the handheld module(115).

Since the forearm (109) does not have an attached sensor module, themovements/orientations of the forearm (109) iscalculated/estimated/predicted from the orientation of the arm module(113) and the orientation of the handheld module (115), as discussedfurther below.

The skeleton model (143) as controlled by the motion processor (145) togenerate inputs for an application (147) running in the computing device(141). For example, the skeleton model (143) can be used to control themovement of an avatar/model of the arm of the user of the computingdevice (141) in a video game, a virtual reality, a mixed reality, oraugmented reality, etc.

In some applications, the handheld module (115) can be replaced with anarm module (113) attached to the hand (105) via holding or via a strap.

Preferably, the arm module (113) has a microcontroller (139) to processthe sensor signals from the IMU (131) of the arm module (113) and acommunication module (133) to transmit the motion/orientation parametersof the arm module (113) to the computing device (141). Similarly, thehandheld module (115) has a microcontroller (129) to process the sensorsignals from the IMU (121) of the handheld module (115) and acommunication module (133) to transmit the motion/orientation parametersof the handheld module (115) to the computing device (141).

Optionally, the arm module (113) and the handheld module (115) have LEDindicators (137 and 127) respectively to indicate the operating statusof the modules (113 and 115).

Optionally, the arm module (113) and the handheld module (115) havehaptic actuators (138 and 128) respectively to provide haptic feedbackto the user via the modules (113 and 115).

Optionally, the handheld module (115) has buttons and other inputdevices (125), such as a touch sensor, a joystick, etc.

Typically, an IMU (e.g., 131 or 121) in a module (e.g., 113 or 115)generates acceleration data from accelerometers, angular velocity datafrom gyrometers/gyroscopes, and/or orientation data from magnetometers.The microcontrollers (139 and 129) perform preprocessing tasks, such asfiltering the sensor data (e.g., blocking sensors that are not used in aspecific application), applying calibration data (e.g., to correct theaverage accumulated error computed by the computing device (141)),transforming motion/position/orientation data in three axes into aquaternion, and packaging the preprocessed results into data packets(e.g., using a data compression technique) for transmitting to the hostcomputing device (141) with a reduced bandwidth requirement and/orcommunication time.

Each of the microcontrollers (129, 139) may include a memory storinginstructions controlling the operation of the respective microcontroller(129 or 139) to perform primary processing of the sensor data from theIMU (121, 131) and control the operations of the communication module(123, 133), and/or other components, such as the LED indicator (127,137), the haptic actuator (128, 138), buttons and other input devices(125).

The computing device (141) may include one or more microprocessors and amemory storing instructions to implement the motion processor (145). Themotion processor (145) may also be implemented via hardware, such asApplication-Specific Integrated Circuit (ASIC) or Field-ProgrammableGate Array (FPGA).

In some instances, one of the modules (113 and 115) is configured as aprimary input device; and the other module is configured as a secondaryinput device that is connected to the computing device (141) via theprimary input device. A secondary input device may use themicroprocessor of its connected primary input device to perform some ofthe preprocessing tasks. A module that communicates directly to thecomputing device (141) is consider a primary input device, even when themodule does not have a secondary input device that is connected to thecomputing device via the primary input device.

In some instances, the computing device (141) specifies the types ofinput data requested, and the conditions and/or frequency of the inputdata; and the modules (113 and 115) report the requested input dataunder the conditions and/or according to the frequency specified by thecomputing device (141). Different reporting frequencies can be specifiedfor different types of input data (e.g., accelerometer measurements,gyroscope/gyrometer measurements, magnetometer measurements, position,orientation, velocity).

In general, the computing device (141) may be a data processing system,such as a mobile phone, a desktop computer, a laptop computer, a headmount virtual reality display, a personal medial player, a tabletcomputer, etc.

FIG. 3 illustrates a skeleton model of an arm. For example, the skeletonmodel of FIG. 3 can be used in the motion processor (145) of FIG. 2 todetermine the orientation of the forearm (109) that does not have anattached sensor module, as illustrated in FIG. 1.

FIG. 3 shows the geometrical representations of the upper arm (101), theforearm (109), and the hand (105) in relation with the elbow joint (103)and the wrist (107) relative to the shoulder (100).

Each of the upper arm (101), the forearm (109), and the hand (105) hasan orientation relative to a common reference system (e.g., the shoulder(100), a room, or a location on the Earth where the user is positioned).The orientation of the upper arm (101), the forearm (109), or the hand(105) can be indicated by a local coordinate system (151, 153, or 155)aligned with the upper arm (101), the forearm (109), or the hand (105).

The orientation of the upper arm (101) and the orientation of the hand(105), as represented by the local coordinate systems (151 and 155) canbe calculated from the motion parameters measured by the IMUs in themodule (113 and 105) attached to the upper arm (101) and the hand (105).

Since the forearm (109) does not have an attached IMU for themeasurement of its orientation, the motion processor (145) uses a set ofassumed relations between the movements of the forearm (109) and thehand (105) to calculate or estimate the orientation of the forearm (109)based on the orientation of the upper arm (101) and the orientation ofthe hand (105), as further discussed below.

FIG. 4 illustrates the determination of the orientation of a forearmaccording to one embodiment.

In FIG. 4, the coordinate system X₁Y₁Z₁ represents the orientation ofthe upper arm (101), where the direction Y₁ is along the lengthwisedirection of the upper arm (101) pointing from the shoulder (100) to theelbow joint (103), as illustrated in FIG. 3. The directions X₁ and Z₁are perpendicular to the direction Y₁. The direction X₁ is parallel tothe direction from the back side of the upper arm (101) to the frontside of the upper arm (101); and the direction X₁ is parallel to thedirection from the inner side of the upper arm (101) to the outer sideof the upper arm (101).

When the arm is in a vertical direction pointing downwards with the hand(105) facing the body of the user, the lengthwise directions Y₁, Y₂, andY₃ of the upper arm (101), the forearm (109), and the hand (105) arealigned with the vertical direction pointing downwards. When in such aposition, the inner sides of the forearm (109) and the upper arm (101)are closest to the body of the user; and the outer sides of the forearm(109) and the upper arm (101) are away from the body of the user; thedirections Z₁, Z₂, and Z₃ of the upper arm (101), the forearm (109), andthe hand (105) are aligned with a direction pointing sideway to theuser; and the directions Z₁, Z₂, and Z₃ of the upper arm (101), theforearm (109), and the hand (105) are aligned with a direction pointingto the front of the user.

Thus, the plane X₁Y₁ is parallel to the direction X₁ from the back sideof the upper arm (101) to the front side of the upper arm (101),parallel to the lengthwise direction Y₁ of the upper arm (101), andperpendicular to the direction Z₁ from the direction from the inner sideof the upper arm (101) to the outer side if the upper arm (101). Thedirection Z₁ coincides with an axis of the elbow joint about which theforearm (109) can rotate to form an angle with the upper arm (101)between their lengthwise directions. When the upper arm (101) isextended in the sideway of the user and in a horizontal position, thedirections X₁ and Z₁ are aligned with (in parallel with) the frontdirection and vertical direction respectively.

The direction Y₂ is aligned with the lengthwise direction of the forearm(109) pointing from the elbow joint (103) to the wrist (107).

The direction Y₃ is aligned with the lengthwise direction of the hand(105) pointing from the wrist (107) towards the fingers.

When the upper arm (101) is extended in the sideway of the user and in ahorizontal position, the directions Y₁, Y₂, and Y₃ coincide with thehorizontal direction pointing the sideway of the user.

When the hand (105) is moved to an orientation illustrated in FIG. 4,the hand (105) can be considered to have moved from the orientation ofthe coordinate system X₁Y₁Z₁ through rotating (165) by an angle γ alongthe lengthwise direction Y₁ and then rotating (161) along the shortestarc (161) such that its lengthwise direction Y₃ arrives at the directionillustrated in FIG. 4. The rotation (161) along the shortest arc (161)corresponding to a rotation of the direction Y₁ by an angle β in a planecontaining both the directions Y₁ and Y₃ along an axis perpendicular toboth the directions Y₁ and Y₃ (i.e., the axis is perpendicular to theplane containing both the directions Y₁ and Y₃).

The projection of the direction Y₃ in the plane X₁Y₁ is assumed to be inthe direction of the lengthwise direction Y₂ of the forearm (109). Theprojection represents a rotation (163) of the direction Y₁ by an angle αin the plane X₁Y₁ along the direction Z₁ according to the shortest arc(163).

It is assumed that the rotation (165) of the hand (105) along itslengthwise direction is a result of the same rotation of the forearm(109) along its lengthwise direction while the forearm (109) isinitially at the orientation aligned with the coordinate system X₁Y₁Z₁.Thus, when the hand (105) has an orientation illustrated in FIG. 4relative to the orientation (X₁Y₁Z₁) of the upper arm (101), theorientation of the forearm (109) is assumed to have moved from theorientation of the coordinate system X₁Y₁Z₁ by rotating (165) along thelengthwise direction Y₁ and then rotating (163) in the plane X₁Y₁ alongthe direction Z₁.

Since the rotations (165, 161 and 163) can be calculated from theorientation of the hand (105) relative to the orientation of the upperarm (101) (e.g., using the orientation data measured by the IMUs of thearm module (113) and the handheld module (115)), the orientation of theforearm (109) can be calculated from the rotations (165 and 163) withoutmeasurement data from an IMU attached to the forearm (109).

After the orientations of the upper arm (101), the forearm (109) and thehand (105) are obtained, the motion processor (145) can compute thepositions of the upper arm (101), the forearm (109) and the hand (105)in a three dimensional space (relative to the shoulder (100)), whichallows the application (147) to present an arm of an avatar in a virtualreality, augmented reality, or mixed reality in accordance with themovement of the arm of the user. The positions of the upper arm (101),the forearm (109) and the hand (105) in a three dimensional space(relative to the shoulder (100)) can also be used to determine thegesture made by the user in the three dimensional space to control theapplication (147).

FIG. 5 shows a method to compute the orientation of a forearm accordingto one embodiment. For example, the method of FIG. 5 can be implementedin a system illustrated in FIG. 2 with an arm module (113) and ahandheld module (115) attached to a hand (105) and an upper arm (101) ina way illustrated FIG. 1 and using the geometrical relations identifiedvia FIG. 3 and FIG. 4.

In FIG. 5, a computing device (141) receives (201) motion data from anarm module (113) identifying an orientation of an upper arm (101) of auser, receives (203) motion data from a handheld module (115)identifying an orientation of a hand (105) of the user, and calculates(203) an orientation of a forearm (109) of the user connecting the hand(105) and the upper arm (101) without a sensor module on the forearm(109).

FIG. 6 shows a detailed method to compute the orientation of a forearmaccording to one embodiment. For example, the method of FIG. 6 can beused to implement the calculation (203) of the orientation of theforearm (109) in the method of FIG. 5.

In FIG. 5, the calculation (203) of the orientation of the forearm (109)is performed by: determining (211) a first rotation β from thelengthwise direction Y₁ of the upper arm (101) to the lengthwisedirection Y₃ of the hand (105), where the first rotation β rotates alonga first axis that is perpendicular to both the lengthwise direction Y₁of the upper arm (101) and the lengthwise direction Y₃ of the hand(105); reversing (213) the first rotation β from the orientation of thehand (105) to obtain a second rotation γ along the lengthwise directionY₁ of the upper arm (101); projecting (215) the lengthwise direction Y₃of the hand (105) to a plane X₁Y₁ that is perpendicular to a second axisZ₁ parallel to a direction from an inner side of the upper arm (101) toan outer side of the upper arm (101) to obtain the lengthwise directionY₂ of the forearm (109); calculating (217) a third rotation α thatrotates within the plane X₁Y₁ and alone the second axis Z₁ from thelengthwise direction Y₁ of the upper arm (101) to the lengthwisedirection Y₂ of the forearm (109); and determining (219) an orientationof the forearm (109) as a result of the orientation X₂Y₂Z₂ of theforearm (109) rotating, from the orientation X₁Y₁Z₁ of the upper arm(101), according to the second rotation γ along the lengthwise directionY₁ of the upper arm (101), and then rotating according to the thirdrotation a in the plane X₁Y₁ and along the second axis Z₁.

As illustrated in FIG. 2, each of the arm module (113) and the handheldmodule (115) is a sensor module that has an inertial measurement unit(IMU) (131, 121) for their orientation measurements. Preferably, thesensor module has a wireless communication device or module (133, 123)for a wireless communication link (117, 119) with the computing device(141). Alternatively, wired connections can be used. The inertialmeasurement unit (IMU) (131, or 121) of the sensor module (113, or 115)may include a micro-electromechanical system (MEMS) gyroscope, amagnetometer, and/or a MEMS accelerometer.

The method of FIG. 6 allows the calculation of theestimated/approximated orientation of the forearm (109) without a needfor a separate inertial measurement unit or sensor module attached tothe forearm (109) of the user, which arrangement reduces the cost of thesystem and improves user experiences.

As an example, the orientation of the forearm can be calculated usingthe following quaternion calculations.

The orientations of the upper arm (101) and the hand (105) can beexpressed as quaternion variables q_(s) and q_(h). The quaternion ofrotation between the upper arm (101) and the hand (105) can becalculated as q_(hs)=q_(s) ⁻¹*q_(h). The lengthwise direction of theupper arm (101) is known for the orientation transformation between theupper arm (101) and the hand (105) (e.g., vector {0; 1; 0} for the lefthand and vector {0; −1; 0} for the right hand). When the lengthwisedirection of the upper arm (101) is expressed as vector o, thelengthwise direction of the hand (105) can be calculated as vectorh=q_(hs)*o*q_(hs) ⁻¹. When the projection of h in the plane containingthe lengthwise direction of the upper arm and the direction from theback of the upper arm to the front of the upper arm is expressed asvector f, the quaternion of rotation (α) along the shortest arc from thevector o to the vector f can be calculated as qzx_(fs). Similarly, thequaternion of rotation (β) along the shortest arc from the vector o tothe vector h can be calculated as qzx_(hs). Since the quaternion ofrotation (γ) along the lengthwise direction of the upper arm (101) isqzx_(hs) ⁻¹*q_(hs), the quaternion of rotation between the upper arm(101) and the forearm (109) is q_(fs)=qzx_(fs)*qzx_(hs) ⁻¹*q_(hs). Thus,the quaternion orientation of the forearm (109) is q_(f)=q_(s)*q_(fs).

The present disclosure includes methods and apparatuses which performthese methods, including data processing systems which perform thesemethods, and computer readable media containing instructions which whenexecuted on data processing systems cause the systems to perform thesemethods.

For example, the computing device (141), the arm module (113) and/or thehandheld module (115) can be implemented using one or more dataprocessing systems.

A typical data processing system may include includes an inter-connect(e.g., bus and system core logic), which interconnects amicroprocessor(s) and memory. The microprocessor is typically coupled tocache memory.

The inter-connect interconnects the microprocessor(s) and the memorytogether and also interconnects them to input/output (I/O) device(s) viaI/O controller(s). I/O devices may include a display device and/orperipheral devices, such as mice, keyboards, modems, network interfaces,printers, scanners, video cameras and other devices known in the art. Inone embodiment, when the data processing system is a server system, someof the I/O devices, such as printers, scanners, mice, and/or keyboards,are optional.

The inter-connect can include one or more buses connected to one anotherthrough various bridges, controllers and/or adapters. In one embodimentthe I/O controllers include a USB (Universal Serial Bus) adapter forcontrolling USB peripherals, and/or an IEEE-1394 bus adapter forcontrolling IEEE-1394 peripherals.

The memory may include one or more of: ROM (Read Only Memory), volatileRAM (Random Access Memory), and non-volatile memory, such as hard drive,flash memory, etc.

Volatile RAM is typically implemented as dynamic RAM (DRAM) whichrequires power continually in order to refresh or maintain the data inthe memory. Non-volatile memory is typically a magnetic hard drive, amagnetic optical drive, an optical drive (e.g., a DVD RAM), or othertype of memory system which maintains data even after power is removedfrom the system. The non-volatile memory may also be a random accessmemory.

The non-volatile memory can be a local device coupled directly to therest of the components in the data processing system. A non-volatilememory that is remote from the system, such as a network storage devicecoupled to the data processing system through a network interface suchas a modem or Ethernet interface, can also be used.

In the present disclosure, some functions and operations are describedas being performed by or caused by software code to simplifydescription. However, such expressions are also used to specify that thefunctions result from execution of the code/instructions by a processor,such as a microprocessor.

Alternatively, or in combination, the functions and operations asdescribed here can be implemented using special purpose circuitry, withor without software instructions, such as using Application-SpecificIntegrated Circuit (ASIC) or Field-Programmable Gate Array (FPGA).Embodiments can be implemented using hardwired circuitry withoutsoftware instructions, or in combination with software instructions.Thus, the techniques are limited neither to any specific combination ofhardware circuitry and software, nor to any particular source for theinstructions executed by the data processing system.

While one embodiment can be implemented in fully functioning computersand computer systems, various embodiments are capable of beingdistributed as a computing product in a variety of forms and are capableof being applied regardless of the particular type of machine orcomputer-readable media used to actually effect the distribution.

At least some aspects disclosed can be embodied, at least in part, insoftware. That is, the techniques may be carried out in a computersystem or other data processing system in response to its processor,such as a microprocessor, executing sequences of instructions containedin a memory, such as ROM, volatile RAM, non-volatile memory, cache or aremote storage device.

Routines executed to implement the embodiments may be implemented aspart of an operating system or a specific application, component,program, object, module or sequence of instructions referred to as“computer programs.” The computer programs typically include one or moreinstructions set at various times in various memory and storage devicesin a computer, and that, when read and executed by one or moreprocessors in a computer, cause the computer to perform operationsnecessary to execute elements involving the various aspects.

A machine readable medium can be used to store software and data whichwhen executed by a data processing system causes the system to performvarious methods. The executable software and data may be stored invarious places including for example ROM, volatile RAM, non-volatilememory and/or cache. Portions of this software and/or data may be storedin any one of these storage devices. Further, the data and instructionscan be obtained from centralized servers or peer to peer networks.Different portions of the data and instructions can be obtained fromdifferent centralized servers and/or peer to peer networks at differenttimes and in different communication sessions or in a same communicationsession. The data and instructions can be obtained in entirety prior tothe execution of the applications. Alternatively, portions of the dataand instructions can be obtained dynamically, just in time, when neededfor execution. Thus, it is not required that the data and instructionsbe on a machine readable medium in entirety at a particular instance oftime.

Examples of computer-readable media include but are not limited tonon-transitory, recordable and non-recordable type media such asvolatile and non-volatile memory devices, read only memory (ROM), randomaccess memory (RAM), flash memory devices, floppy and other removabledisks, magnetic disk storage media, optical storage media (e.g., CompactDisk Read-Only Memory (CD ROM), Digital Versatile Disks (DVDs), etc.),among others. The computer-readable media may store the instructions.

The instructions may also be embodied in digital and analogcommunication links for electrical, optical, acoustical or other formsof propagated signals, such as carrier waves, infrared signals, digitalsignals, etc. However, propagated signals, such as carrier waves,infrared signals, digital signals, etc. are not tangible machinereadable medium and are not configured to store instructions.

In general, a machine readable medium includes any mechanism thatprovides (i.e., stores and/or transmits) information in a formaccessible by a machine (e.g., a computer, network device, personaldigital assistant, manufacturing tool, any device with a set of one ormore processors, etc.).

In various embodiments, hardwired circuitry may be used in combinationwith software instructions to implement the techniques. Thus, thetechniques are neither limited to any specific combination of hardwarecircuitry and software nor to any particular source for the instructionsexecuted by the data processing system.

In the foregoing specification, the disclosure has been described withreference to specific exemplary embodiments thereof. It will be evidentthat various modifications may be made thereto without departing fromthe broader spirit and scope as set forth in the following claims. Thespecification and drawings are, accordingly, to be regarded in anillustrative sense rather than a restrictive sense.

What is claimed is:
 1. A system, comprising: a first sensor module having an inertial measurement unit and attached to an upper arm of a user, the first sensor module generating first motion data identifying an orientation of the upper arm; a second sensor module having an inertial measurement unit and attached to a hand of the user, the second sensor module generating second motion data identifying an orientation of the hand; and a computing device coupled to the first sensor module and the second sensor module through communication links, the computing device calculating, based on the orientation of the upper arm and the orientation of the hand, an orientation of a forearm connected to the hand by a wrist of the user and connected to the upper arm by an elbow joint of the user.
 2. The system of claim 1, wherein each of the first sensor module and the second sensor module further includes a communication device for a communication link with the computing device.
 3. The system of claim 1, wherein each of the inertial measurement unit of the first sensor module and the inertial measurement of the second sensor module includes a micro-electromechanical system (MEMS) gyroscope.
 4. The system of claim 3, wherein each of the inertial measurement unit of the first sensor module and the inertial measurement of the second sensor module further includes a magnetometer and a MEMS accelerometer.
 5. The system of claim 1, wherein the orientation of the forearm is calculated without an inertial measurement unit attached to the forearm of the user.
 6. The system of claim 1, wherein the orientation of the forearm is calculated without a sensor module on the forearm of the user.
 7. The system of claim 1, wherein the computing device calculates the orientation of the forearm by: determining a first rotation from a lengthwise direction of the upper arm to a lengthwise direction of the hand rotating along a first axis that is perpendicular to both the lengthwise direction of the upper arm and the lengthwise direction of the hand; reversing the first rotation along the first axis from the orientation of the hand to obtain a second rotation along the lengthwise direction of the upper arm; projecting, on to a plane, the lengthwise direction of the hand to obtain a lengthwise direction of the forearm; calculating a third rotation from the lengthwise direction of the upper arm to the lengthwise direction of the forearm along a second axis perpendicular to the plane; and determining, by the computing device, the orientation of the forearm as a combination of an initial orientation aligned with the upper arm rotated according to the second rotation along the lengthwise direction of the upper arm and then rotated according to the third rotation along the second axis.
 8. The system of claim 7, wherein the second axis is parallel to a direction from an inner side of the upper arm to an outer side of the upper arm; and the plane is parallel with the lengthwise direction of the upper arm and a direction from a back side of the upper arm to a front side of the upper arm.
 9. A method, comprising: receiving, from a first sensor module attached to an upper arm of a user, first motion data identifying an orientation of the upper arm; receiving, from a second sensor module attached to a hand of the user, second motion data identifying an orientation of the hand; calculating, based on the orientation of the upper arm and the orientation of the hand, an orientation of a forearm connected to the hand by a wrist of the user and connected to the upper arm by an elbow joint of the user; and determining a position of the hand in a three dimensional space based on the orientation of the upper arm, and the orientation of the forearm, the orientation of the hand, and a skeleton model of the upper arm, the forearm, and the hand.
 10. The method of claim 9, wherein the orientation of the forearm is calculated to have a rotation of the forearm about a lengthwise direction of the forearm that is equal to a first angle of a rotation of the hand about a lengthwise direction of the hand.
 11. The method of claim 10, wherein the orientation of the forearm is calculated to have a rotation, in a plane, of the lengthwise direction of the forearm from a lengthwise direction of the upper arm that is equal to a second angle of a rotation, projected in the plane, of the lengthwise direction of the hand from the lengthwise direction of the upper arm in the plane.
 12. The method of claim 11, wherein the plane is perpendicular to a direction pointing from an inner side of the upper arm and an outer side of the upper arm.
 13. The method of claim 12, wherein the plane is parallel to the lengthwise direction of the upper arm.
 14. The method of claim 13, wherein the plane is further parallel to a direction pointing from a back side of the upper arm to a front side of the upper arm.
 15. The method of claim 11, wherein the orientation of the forearm is calculated to be equal to a first orientation resulting from: rotating, from a second orientation aligned with the orientation of the upper arm, about the lengthwise direction of the forearm, by the first angle to arrive a third orientation; and rotating, from the third orientation, about an axis parallel to a direction from a back side of the upper arm to a front side of the upper arm, by the second angle to arrive at the first orientation.
 16. A non-transitory computer storage medium storing instructions which, when executed by a computing device, instructs the computing device to perform a method, the method comprising: receiving, by the computing device from a first sensor module attached to an upper arm of a user, first motion data identifying an orientation of the upper arm; receiving, by the computing device from a second sensor module attached to a hand of the user, second motion data identifying an orientation of the hand; and calculating, by the computing device based on the orientation of the upper arm and the orientation of the hand, an orientation of a forearm connected to the hand by a wrist of the user and connected to the upper arm by an elbow joint of the user.
 17. The non-transitory computer storage medium of claim 16, wherein the calculating comprises: calculating, by the computing device, an orientation of the hand relative to the upper arm; determining, by the computing device, a first rotation of a lengthwise direction of the hand from a lengthwise direction of the upper arm along a closest arc to the lengthwise direction of the hand in the orientation of the hand relative to the upper arm; subtracting, by the computing device, the first rotation from the orientation of the hand relative to the upper arm to obtain a second rotation of the hand along the lengthwise direction of the upper arm; projecting, on to a plane by the computing device, the lengthwise direction of the hand in the orientation of the hand relative to the upper arm to determine a lengthwise direction of the forearm relative to the upper arm; computing, by the computing device, a closest arc rotation from the lengthwise direction of the upper arm to the lengthwise direction of the forearm relative to the upper arm; and determining, by the computing device, the orientation of the forearm based on combination a rotation from an orientation aligned with the upper arm, along the lengthwise direction of the upper arm according to the second rotation of the hand along the lengthwise direction of the upper arm and a rotation within the plane according to the closest arc rotation from the lengthwise direction of the upper arm to the lengthwise direction of the forearm relative to the upper arm.
 18. The non-transitory computer storage medium of claim 17, wherein the plane is perpendicular to a direction pointing from an inner side of the upper arm and an outer side of the upper arm.
 19. The non-transitory computer storage medium of claim 18, wherein the plane is parallel to the lengthwise direction of the upper arm.
 20. The non-transitory computer storage medium of claim 19, wherein the plane is further parallel to a direction pointing from a back side of the upper arm to a front side of the upper arm. 